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ABSTRACT 

Modern applications often operate on data in multiple administra- 
tive domains. In this federated setting, participants may not fully 
trust each other. These distributed applications use transactions as 
a core mechanism for ensuring reliability and consistency with per- 
sistent data. However, the coordination mechanisms needed for 
transactions can both leak confidential information and allow unau- 
thorized influence. 

By implementing a simple attack, we show these side channels 
can be exploited. However, our focus is on preventing such attacks. 
We explore secure scheduling of atomic, serializable transactions 
in a federated setting. While we prove that no protocol can guaran- 
tee security and liveness in all settings, we establish conditions for 
sets of transactions that can safely complete under secure schedul- 
ing. Based on these conditions, we introduce staged commit , a 
secure scheduling protocol for federated transactions. This proto- 
col avoids insecure information channels by dividing transactions 
into distinct stages. We implement a compiler that statically checks 
code to ensure it meets our conditions, and a system that schedules 
these transactions using the staged commit protocol. Experiments 
on this implementation demonstrate that realistic federated transac- 
tions can be scheduled securely, atomically, and efficiently. 

1. INTRODUCTION 

Many modem applications are distributed, operating over data 
from multiple domains. Distributed protocols are used by applica- 
tions to coordinate across physically separate locations, especially 
to maintain data consistency. However, distributed protocols can 
leak confidential information unless carefully designed otherwise. 

Distributed applications are often structured in terms of transac- 
tions, which are atomic groups of operations. For example, when 
ordering a book online, one or more transactions occur to ensure 
that the same book is not sold twice, and to ensure that the sale of 
a book and payment transfer happen atomically. Transactions are 
ubiquitous in modern distributed systems. Implementations include 
Google’s Spanner [14], Postgres [35], and Microsoft’s Azure Stor- 
age [11]. Common middleware such as Enterprise Java Beans [32] 
and Microsoft .NET [1] also support transactions. 

Many such transactions are distributed, involving multiple au- 
tonomous participants (vendors, banks, etc.). Crucially, these par- 
ticipants may not be equally trusted with all data. Standards such as 
X/Open XA [2] aim specifically to facilitate transactions that span 
multiple systems, but none address information leaks inherent to 
transaction scheduling. 

Distributed transaction implementations are often based on the 
two-phase commit protocol (2PC) [20]. We show that 2PC can cre- 
ate unintentional channels through which private information may 
be leaked, and trusted information may be manipulated. We expect 
our results apply to other protocols as well. 

There is a fundamental tension between providing strong consis- 


tency guarantees in an application and respecting the security re- 
quirements of the application’s trust domains. This work deepens 
the understanding of this trade-off and demonstrates that providing 
both strong consistency and security guarantees, while not always 
possible, is not a lost cause. 

Concretely, we make the following contributions in this paper: 

• We describe abort channels , a new kind of side channel through 
which confidential information can be leaked in transactional 
systems (§2). 

• We demonstrate exploitation of abort channels on a distributed 
system (§2.3). 

• We define an abstract model of distributed systems, trans- 
actions, and information flow security (§3), and introduce 
relaxed observational determinism, a noninterference-based 
security model for distributed systems (§3.7.2). 

• We establish that within this model, it is not possible for any 
protocol to securely serialize all sets of transactions, even if 
the transactions are individually secure (§4). 

• We introduce and prove a sufficient condition for ensuring 
serializable transactions can be securely scheduled (§5). 

• We define the staged commit protocol, a novel secure schedul- 
ing protocol for transactions meeting this condition (§6). 

• We implement our novel protocol in the Fabric system [30], 
and extend the Fabric language and compiler to statically en- 
sure transactions will be securely scheduled (§7). 

• We evaluate the expressiveness of the new static checking 
discipline and the runtime overhead of the staged commit 
protocol (§8). 

We discuss related work further in §9, and conclude in § 10. 

2. ABORT CHANNELS 

Two transactions working with the same data can conflict if at 
least one of them is writing to the data. Typically, this means that 
one (or both) of the transactions has failed and must be aborted. 

In many transaction protocols, including 2PC, a participant 1 in- 
volved in both transactions can abort a failed transaction by send- 
ing an abort message to all other participants in the failed transac- 
tion [20]. These abort messages can create unintended abort chan- 
nels, through which private information can be leaked, and trusted 
information can be manipulated. 

An abort message can convey secret information if a participant 
aborts a transaction otherwise likely to be scheduled, because an- 
other participant in the same transaction might deduce something 

transaction participants are often processes or network nodes. 
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Figure 1: Rainforest example. Gloria and Fred each buy an Outel 
chip via Rainforest’s store. Gloria’s transaction is in red, dashed 
arrows; Fred’s is in blue, solid arrows. 

about the aborting participant. For example, that other partici- 
pant might guess that the abort is likely caused by the presence 
of another — possibly secret — conflicting transaction. 

Conspirators might deliberately use abort channels to covertly 
transfer information within a system otherwise believed to be se- 
cure. Although abort channels communicate at most one bit per 
(attempted) transaction, they could be used as a high-bandwidth 
covert channel for exfiltration of sensitive information. Current 
transactional systems can schedule over 100 million transactions 
per second, even at modest system sizes [18]. It is difficult to know 
if abort channels are already being exploited in real systems, but 
large-scale, multi-user transactional systems such as Spanner [14] 
or Azure Storage [11] are in principle vulnerable. 

Abort messages also affect the integrity of transaction schedul- 
ing. An abort typically causes a transaction not to be scheduled. 
Even if the system simply retries the transaction until it is sched- 
uled, this still permits a participant to control the ordering of trans- 
actions, even if it has no authority to affect them. For example, 
a participant might gain some advantage by ensuring that its own 
transactions always happen after a competitor’s. 

Transactions can also create channels that leak information based 
on timing or termination [6, 10]. We treat timing and termination 
channels as outside the scope of this work, to be handled by mecha- 
nisms such as timing channel mitigation [26, 5, 9]. Abort channels 
differ from these previously identified channels in that information 
leaks via the existence of explicit messages, with no reliance on 
timing other than their ordering. Timing mitigation does not con- 
trol abort channels. 

2.1 Rainforest Example 

A simple example illustrates how transaction aborts create a chan- 
nel that can leak information. Consider a web- store application for 
the fictional on-line retailer Rainforest, illustrated in Fig. 1. Rain- 
forest’s business operates on data from suppliers, customers, and 
banks. Rainforest wants to ensure that it takes money from cus- 
tomers only if the items ordered have been shipped from the sup- 
pliers. As a result, Rainforest implements purchasing using serial- 
izable transactions. Customers expect that their activities do not in- 
fluence each other, and that their financial information is not leaked 
to suppliers. These expectations might be backed by law. 

In Fig. 1, Gloria and Fred are both making purchases on Rain- 
forest at roughly the same time. They each purchase an Outel chip, 
and pay using their accounts at CountriBank. 

If Rainforest uses 2PC to perform both of these transactions, it is 
possible for Gloria to see an abort when Outel tries to schedule her 
transaction and Fred’s. The abort leaks information about Fred’s 
purchase at Outel to Gloria. Alternatively, if Gloria is simultane- 
ously using her bank account in an unrelated purchase, scheduling 
conflicts at the bank might leak to Outel, which could thereby learn 
of Gloria’s unrelated purchase. 



Figure 2: The events of the transactions in Fig. 1. Gloria’s trans- 
action consists of ro, ri, r 2 , r 3 , r 4 , and rs. Bob’s consists of bo, 
bi, b 2 , b 3 , b 4 , and bs. Happens-before (— >) relationships are ar- 
rows. The shaded blocks around events indicate locations, and are 
labeled with participants from Fig. 1. 

These concerns are about confidentiality, but transactions may 
also create integrity concerns. The bank might choose to abort 
transactions to affect the order in which Outel sells chips. Rain- 
forest and Outel may not want the bank to have this power. 

2.2 Hospital Example 

As a second, running example, we use two small programs with 
an abort channel. Suppose Patsy is a trusted hospital employee, 
running the code in Fig. 3a to collect the addresses of HIV-positive 
patients in order to send treatment reminders. Patsy runs her trans- 
action on her own computer, which she fully controls, but it in- 
teracts with a trusted hospital database on another machine. Patsy 
starts a transaction for each patient p, where transaction blocks are 
indicated by the keyword atomic. If p does not have HIV, the 
transaction finishes immediately. Fig. 3c shows the resulting trans- 
action in solid blue. (Events in the transaction are represented as 
ovals; arrows represent dependencies between transaction events.) 
Otherwise, if the patient has HIV, Patsy’s transaction reads the pa- 
tient’s address and prints it (the blue transaction in Fig. 3c, includ- 
ing dashed events). 

Suppose Mallory is another employee at the same hospital, but is 
not trusted to know each patient’s HIV status. Mallory is, however, 
trusted with patient addresses. Like Patsy, Mallory’s code runs on 
her own computer, which she fully controls, but interacts with the 
trusted hospital database on another machine. She runs the code in 
Fig. 3b to update each patient’s address in a separate transaction, 
resulting in the red transaction in Fig. 3c. When Mallory updates 
the address of an HIV-positive patient, her transaction might con- 
flict with one of Patsy’s, and Mallory would observe an abort. Thus 
Mallory can learn which patients are HIV-positive by updating each 
patient’s address while Patsy is checking the patients’ HIV statuses. 
Each time one of Mallory’s transactions aborts, private information 
leaks: that patient has HIV. 

One solution to this problem is to change Patsy’s transaction: 
instead of reading the address only if the patient is HIV positive, 
Patsy reads every patient’s address. This illustrates a core goal of 
our work: identifying which programs can be scheduled securely. 
In Fig. 4a, lines 3 and 4 of Patsy’s code have been switched. As 
Fig. 4c shows, both possible transactions read the patient’s address. 
Since Mallory cannot distinguish which of Patsy’s transactions has 
run, she cannot learn which patients have HIV. 

2.3 Attack Demonstration 

Using code resembling Fig. 3, we implemented the attack de- 
scribed in our hospital example (§2.2) using the Fabric distributed 
system [29, 30]. We ran nodes representing Patsy and Mallory, and 
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1 atomic { 

2 h = p.hasHiv; 

3 if (h) { 

4 x = p. address; 

5 print(x); 

6 } 

7 } 

(a) Patsy’s code 


A ■ Print address^ 

Blr ^ 

x / High Security (H) 

> i_ 


/ Low Security (L) 

1 atomic { 

2 p.address+='L" ; 

3 } 

(b) Mallory’s code 

(c) Resulting transactions 


Figure 3: Insecure hospital scenario. Patsy runs a program (3a) 
for each patient p. If p has HIV (which is private information), 
she prints out p’s address for her records. The resulting transaction 
takes one of two forms. Both begin with the event Patsy start. If p 
is HIV negative, the transaction ends with Read HIV. Otherwise, it 
includes the blue events with dashed outlines. Meanwhile, Mallory 
updates the p’s (less secret) address (3b), resulting in the transac- 
tion with red, solid-bordered events. This conflicts with Patsy’s 
transaction, requiring the system to order the update and the read, 
exactly when p has HIV (“?” in 3c). 


a storage node for the patient records. 

To improve the likelihood of Mallory conflicting with Patsy (and 
thereby receiving an abort), we had Patsy loop roughly once a sec- 
ond, continually reading the address of a single patient after veri- 
fying their HIV-positive status. Meanwhile, Mallory attempted to 
update the patient’s address with approximately the same frequency 
as Patsy’s transaction. 

Like many other distributed transaction systems, Fabric uses two- 
phase commit. Mallory’s window of opportunity for receiving an 
abort exists between the two phases of Patsy’s commit, which ordi- 
narily involves a network round trip. However, both nodes were run 
on a single computer. To model a cloud-based server, we simulated 
a 100 ms network delay between Patsy and the storage node. 

Getting this to work was challenging, because Fabric caches its 
objects optimistically. When Mallory updates the patient’s address, 
it would invalidate Patsy’s cached copy, causing Patsy’s next trans- 
action to abort and retry. Furthermore, Fabric implements an expo- 
nential back-off algorithm for retrying aborted transactions. As a 
result, we had to carefully tune the transaction frequencies to pre- 
vent Mallory from starving out Patsy. 

We ran this experiment for 90 minutes. During this time, Mal- 
lory received an abort roughly once for every 20 transactions Patsy 
attempted. As a result, approximately every 20 seconds, Mallory 
learned that a patient had HIV. In principle, many such attacks 
could be run in parallel, so this should be seen as a minimal, rather 
than a maximal, rate of information leakage for this setup. 

As described later, our modified Fabric compiler (§ 7) correctly 
rejects Patsy’s code. We amended Patsy’s code to reflect Fig. 4, and 
our implementation of the staged commit protocol (§6) was able 
to schedule the transactions without leaking information. Mallory 
was no more or less likely to receive aborts regardless of whether 
the patient had HIV. 

3. SYSTEM MODEL 

We introduce a formal, abstract system model that serves as our 
framework for developing protocols and proving their security prop- 
erties. Despite its simplicity, the model captures the necessary fea- 
tures of distributed transaction systems and protocols. As part of 
this model, we define what it means for transactions to be serializ- 
able and what it means for a protocol to serialize transactions both 
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4 if (h) { 

5 print(x); 
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Figure 4: Secure hospital scenario. A secure version of Fig. 3, 
in which lines 3 and 4 of Patsy’s code (3a) are switched, and the 
resulting lines 2 and 3 can be run in parallel (( || )). Thus the trans- 
action reads p’s address regardless of whether p has HIV, and so 
Mallory cannot distinguish which form Patsy’s transaction takes. 

correctly and securely. 

3.1 State and Events 

Similarly to Lamport [27], we define a system state to include 
a finite set of events , representing a history of the system up to a 
moment in time. An event (denoted e) is an atomic native action 
that takes place at a location , which can be thought of as a physical 
computer on the network. Some events may represent read oper- 
ations (“the variable x had the value 3”), or write operations (“2 
was written into the variable y”). In Figures 3 and 4, for example, 
events are represented as ovals, and correspond to lines of code. 

Also part of the system state is a causal ordering on events. Like 
Lamport’s causality [27], the ordering describes when one event e\ 
causes another event e<i. In this case, we say e± happens before e 2 , 
written as ei^e2. This relationship would hold if, for example, 
ei is the sending of a message, and e2 its receipt. The ordering 
(— >) is a strict partial order: irreflexive, asymmetric, and transitive. 
Therefore, ei^e 2 and e 2 ^e 3 together imply ei^>e 3 . 

The arrows in Figures 2 to 4 show happens-before relationships 
for the transactions involved. 

3.2 Information Flow Lattice 

We extend Lamport’s model by assigning to each event e a se- 
curity label , written £(e), which defines the confidentiality and in- 
tegrity requirements of the event. Events are the most fine-grained 
unit of information in our model, so there is no distinction between 
the confidentiality of an event’s occurrence and that of its contents. 
Labels in our model are similar to high and low event sets [37, 
13]. In Figures 3 and 4, two security labels, High and Low (H and 
L for short), are represented by the events’ positions relative to the 
dashed line. 

For generality, we assume that labels are drawn from a lattice [15], 
depicted in Fig. 5. Information is only permitted to flow upward in 
the lattice. We write “£(ei) is below £(e 2 )” as £(ei)0£(e 2 ), mean- 
ing it is secure for the information in e\ to flow to e2. 

For instance, in Fig. 3, information should not flow from any 
events labeled H to any labeled L. Intuitively, we don’t want secret 
information to determine any non-secret events, because unautho- 
rized parties might learn something secret. However, information 
can flow in the reverse direction: reading the patient’s address (la- 
beled L) can affect Patsy’s printout (labeled H): L C H. 

The join (U) of two labels represents their least upper bound: 
£iE(£il_l£ 2 ) and The meet (n) of two labels repre- 

sents their greatest lower bound: (£ 1 n£ 2 )Q£i and (£i\i£ 2 )Q£ 2 . 

Like events, each location has a label, representing a limit on 
events with which that location can be trusted. No event should 
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Figure 5: Security lattice: The dot represents a label in the lattice, 
and the dashed lines divide the lattice into four quadrants relative 
to this label. If the label represents an event, then only events with 
labels in quadrant B may influence this event, and this event may 
only influence events with labels in quadrant A. If the label rep- 
resents a location, then only events with labels in quadrant C may 
occur at that location. 


have more integrity than its location. Similarly, no event should 
be too secret for its location to know. Thus, in Fig. 5, only events 
to the left of a. location’s label (i.e., region C in the figure) may take 
place at that location. 

For example, consider Gloria’s payment event at CountriBank in 
the Rainforest example Fig. 1. This event (rs in Fig. 2) represents 
money moving from Gloria’s account to Outel’s. The label i of r$ 
should not have any more integrity than CountriBank itself, since 
the bank controls r 5 . Likewise, the bank knows about r 5 , so i 
cannot be more confidential than the CountriBank’ s label. This 
would put i to the left of the label representing CountriBank in the 
lattice of Fig. 5. 

Our prototype implementation of secure transactions is built us- 
ing the Fabric system [30], so the lattice used in the implementation 
is based on the Decentralized Label Model (DLM) [33]. However, 
the results of this paper are independent of the lattice used. 

3.3 Conflicts 

Two events in different transactions may conflict. This is a prop- 
erty inherent to some pairs of events. Intuitively, conflicting events 
are events that must be ordered for data to be consistent. For exam- 
ple, if e\ represents reading variable x , and e 2 represents writing 
x , then they conflict, and furthermore, the value read and the value 
written establish an ordering between the events. Likewise, if two 
events both write variable x , they conflict, and the system must 
decide their ordering because it affects future reads of x. 

In our hospital example (Figures 3 and 4), the events Read ad- 
dress and Update address conflict. Specifically, the value read will 
change depending on whether it is read before or after the update. 
Thus for any such pair of events, there is a happens-before (— >) 
ordering between them, in one direction or the other. 

We assume that conflicting events have the same label. This as- 
sumption is intuitive in the case of events that are reads and writes 
to the same variable (that is, storage location). Read and write op- 
erations in separate transactions could have occurred in either or- 
der, so the happens-before relationship between the read and write 
events cannot be predicted in advance. 

Our notion of conflict is meant to describe direct interaction be- 
tween transactions. Hence, we also assume any conflicting events 
happen at the same location. 



Figure 6: An example system state. The events r 0 , ri , and r 2 form 
transaction R, and the events bo, bi, and b 2 form transaction B. 
Event p is not part of either transaction. It may be an input, such 
as a network delay event, or part of a protocol used to schedule 
the transactions. In this state, ri^p — >bi, which means that ri 
happens before bi, and so the transactions are ordered: R^>B. 

3.4 Serializability and Secure Information Flow 

Traditionally a transaction is modeled as a set of reads and writes 
to different objects [34]. We take a more abstract view, and model a 
transaction as a set of events that arise from running a piece of code. 
Each transaction features a start event , representing the decision to 
execute the transaction’s code. Start events, by definition, happen 
before all others in the transaction. Multiple possible transactions 
can feature the same start event: the complete behavior of the trans- 
action’s code is not always determined when it starts executing, and 
may depend on past system events. 

Fig. 4c shows two possible transactions, in blue, that can result 
from running the secure version of Patsy’s code. They share the 
three events in solid blue, including the start event (Patsy start); 
one transaction contains a fourth event, Print address. The figure 
also shows in red the transaction resulting from Mallory’s code. 
Fig. 6 is a more abstract example, in which r 0 is the start event of 
transaction R, and b 0 is the start event of transaction B. 

In order to discuss what it means to serialize transactions, we 
need a notion of the order in which transactions happen. We obtain 
this ordering by lifting the happens-before relation on events to a 
happens-before (— >) relation for transactions. We say that transac- 
tion T 2 directly depends on Ti, written T\ -A T 2 , if an event in T\ 
happens before an event in T 2 : 

Ti -< T 2 = Ti T 2 A 3e± E Ti, e 2 E T 2 . e±— >e 2 

The happens-before relation on transactions (— >) is the transitive 
closure of this direct dependence relation -<. Thus, in Fig. 6, the 
ordering R^>B holds. Likewise, Fig. 7 is a system state featur- 
ing the transactions from our hospital example (Fig. 4), in which 
Patsy ^>Mallory holds. 

Def. 1 (Serializability). Transactions are serializable ex- 
actly when happens-before is a strict partial order on transactions. 

Any total order consistent with this strict partial order would then 
respect the happens-before ordering (— >) of events. Such a total 
ordering would represent a serial order of transactions. 

Def. 2 (Secure Information Flow). A transaction is in- 
formation-flow secure if happens-before (— >) relationships between 
transaction events — and therefore causality — are consistent with 
permitted information flow: 

ei—>e2 => ^(ei)C£(e 2 ) 

This definition represents traditional, conservative information 
flow control within each transaction. Intuitively, each transaction 
itself cannot cause a security breach (although this definition says 
nothing about the protocol scheduling them). In our hospital exam- 
ple, Patsy’s transaction in Fig. 3c is not information-flow secure , 
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Figure 7 : A possible system state after running transactions from 
Fig. 4c, assuming the patient has HIV, and an exclusive lock is used 
to order the transactions. (Events prior to everything in both trans- 
actions are not shown.) Because Patsy acquires the lock first, the 
transactions are ordered Patsy^Mallory. While each transaction 
is information-flow secure (a property of events within a transac- 
tion), when Patsy releases the lock after her transaction, a high se- 
curity event happens before a low security one. We discuss secure 
scheduling protcols in § 6. 


since Read HIV happens before Read address, and yet the label of 
Read HIV, H, does not flow to the label of Read address, L. How- 
ever, in the modified, secure version (Fig. 4c), there are no such 
insecure happens-before relationships, so Patsy’s transaction is se- 
cure. 


3.5 Network and Timing 

Although this model abstracts over networks and messaging, we 
consider a message to comprise both a send event and a receive 
event. We assume asynchronous messaging: no guarantees can be 
made about network delay. Perhaps because this popular assump- 
tion is so daunting, many security researchers ignore timing-based 
attacks. There are methods for mitigating leakage via timing chan- 
nels [26, 5, 9] but in this work we too ignore timing. 

To model nondeterministic message delay, we introduce a net- 
work delay event for each message receipt event, with the same 
label and location. The network delay event may occur at any time 
after the message send event. It must happen before (— >) the cor- 
responding receipt event. In Fig. 6, event 1*1 could represent send- 
ing a message, event p could be the corresponding network delay 
event, which is not part of any transaction, and event bi could be 
the message receipt event. Fig. 6 does not require p to be a network 
delay event. It could be any event that is not part of either transac- 
tion. For example, it might be part of some scheduling protocol. 

3.6 Executions, Protocols, and Inputs 

An execution is a start state paired with a totally ordered se- 
quence of events that occur after the start state. This sequence must 
be consistent with happens-before (— >). Recall that a system state 
is a set of events (§3.1). Each event in the sequence therefore cor- 
responds to a system state containing all the events in the start state, 


and all events up to and including this event in the sequence. View- 
ing an execution as a sequence of system states, an event is sched- 
uled if it is in a state, and once it is scheduled, it will be scheduled 
in all later states. Two executions are equivalent if their start states 
are equal, and their sequences contain the same set of events, so 
they finish with equal system states (same set of events, same — >). 
A full execution represents the entire lifetime of the system, so its 
start state contains no events. 

For example, Fig. 8 illustrates two equivalent full executions 
ending in the system state from Fig. 6. 

A transaction scheduling protocol determines the order in which 
each location schedules the events of transactions. Given a set of 
possible transactions, a location, and a set of events representing 
a system state at that location, a protocol decides which event is 
scheduled next by the location: 

protocol : set (Transactions) x Location x State — >• event 

Protocols can schedule an event from a started (but unfinished) 
transaction, or other events used by the protocol itself. In order to 
schedule transaction events in ways that satisfy certain constraints, 
like serializability, protocols may have to schedule additional events, 
which are not part of any transaction. These can include message 
send and receipt events. For example, in Fig. 7, the locking events 
are not part of any transaction, but are scheduled by the protocol in 
order to ensure serializability. 

Certain kinds of events are not scheduled by protocols, because 
they are not under the control of the system. Events representing 
external inputs, including the start events of transactions, can hap- 
pen at any time: they are fundamentally nondeterministic. We also 
treat the receive times of messages as external inputs. Each mes- 
sage receive event is the deterministic result of its send event and 
of a nondeterministic network delay event featuring the same secu- 
rity label as the receive event. We refer to start and network delay 
events collectively as nondeterministic input events (NIEs). 

Protocols do not output NIEs. Instead, an NIE may appear at 
any point in an execution, and any prior events in the execution 
can happen before (— >) the NIE. Recall that an execution features 
a sequence of events, each of which can be seen as a system state 
featuring all events up to that point. An execution E is consistent 
with a protocol p if every event in the sequence is either an NIE, or 
the result of p applied to the previous state at the event’s location. 
We sometimes say p results in E to mean “ E is consistent with p.” 

As an example, assume all events in Fig. 6 have the same location 
L, and no messages are involved. Start events 1*0 and bo are NIEs. 
Every other event has been scheduled by a protocol. Fig. 8 shows 
two different executions, which may be using different protocols, 
determining which events to schedule in each state. We can see that 
in the top execution of Fig. 8, the protocol maps: 

{R,B,...},L,{r 0 } ^ 1*1 
{R, B, . . .}, L, {r 0 , 1*1} r 2 

{R, B, . . .}, L, {r 0 , 1*1, r 2 , b 0 } i-)> p 
{R, B, . . .}, L, {r 0 , ri, r 2 , b 0 , p} 1— »• bi 
{R, B,...},L, {r 0 ,ri,r 2 ,b 0 ,p,bi} b 2 

The protocol in the bottom execution of Fig. 8 maps: 

{R? B, . . .}, L, {r 0 , b 0 } h- >> n 
{R, B, . . .}, L, {r 0 , b 0 , 1*1} !->• p 
{R, B, . . .}, L, {r 0 , b 0 , 1*1, p} 1— »• bi 
{R? B, . . .}, L, {r 0 , b 0 , 1*1 , p, bi} h- >> b 2 
{R, B, . . L, {r 0 , b 0 , ri, p, bi, b 2 } ^ r 2 

Ultimately, a protocol must determine the ordering of transac- 
tions. If the exact set of start events to be scheduled (as opposed 
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Figure 8: Two equivalent full executions for the system state from Fig. 6. Each begins with a start state (the empty set for full executions), 
followed by a sequence of events, each of which corresponds to the resulting system state. 


to start events possible) were always known in advance, scheduling 
would be trivial. A protocol should not require one transaction to 
run before another a priori : start events from any subset of pos- 
sible transactions may be scheduled at any time. No protocol 
should result in a system state in which such a start event cannot be 
scheduled, or an incomplete transaction can never finish. 

3.7 Semantic Security Properties 

Consider an observer who can only “see” events at some security 
level i or below. If two states Si and S 2 are indistinguishable to the 
observer, then after a program runs, noninterference requires that 
the resulting executions remain indistinguishable to the observer. 
Secret values, which the observer cannot see, may differ in Si and 
S 2 , and may result in different states at the end of the executions, 
but the observer should not be able to see these differences. 

3. 7 . 1 Possibilistic Noninterference 

David Sutherland’s hyperproperty Generalized Noninterference 2 [43] 
generalizes Goguen and Meseguer’s noninterference [22]. His model 
features “possible execution sequences”, much like our executions , 
each of which is a sequence of system states. For a given observer, 
some information is low observable, meaning the observer may 
learn it. Other information is high, meaning it’s too secret for the 
observer to know. His model also features some events, called “sig- 
nals,” representing inputs , which can be either low or high. Possi- 
bilistic Noninterference, then, requires that for any given execution 
Ei , it must be possible to change the high inputs of E\ to those of 
any other valid execution E 2 , and create a valid, possible execution 
E 3 without changing any low events: 

Highjinputs (E 3 ) = Highjinputs (E^A 
15 2 ' 3 ' Low_events(E 3 ) = Low_events(Ei ) 

In a sense, an observer can’t make any observations that change 
the possible set of high inputs, but might be able to infer which are 
probable. This is recognized as a fairly weak form of noninterfer- 
ence in nondeterministic systems. [13] 

In our hospital example, as illustrated in Fig. 4, the system de- 
termines which of Patsy’s transactions will run based upon whether 
p.hasHivistrue. We can treat this condition to be a high- security 
event that happens before all reads of p.hasHiv. If we classify 
this past high-security event as input, and all low-security events as 
low-observable for Mallory, then we must ensure that when Patsy’s 
code runs, the set of possible low-security events that result is the 
same regardless of whether p.hasHiv. Patsy’s possible transac- 
tions in Fig. 4 ensure possibilistic noninterference, while her trans- 
actions in Fig. 3 do not, since whether or not Read address occurs 
depends on p . hasHiv. 

3. 7.2 Relaxed Observational Determinism 


2 McCullough coins the term “Generalized Noninterference” [31], 

and Clarkson and Schneider define hyperproperties [13]. 


Semantic conditions for information security are typically based 
on some variant of noninterference [22, 38]. These variants are 
often distinguished by their approaches to nondeterminism. How- 
ever, many of these semantic security conditions fail under refine- 
ment: if some nondeterministic choices are fixed, security is vio- 
lated [46]. However, low-security observational determinism [37, 
46] is a strong property that is secure under refinement: intuitively, 
if an observer with label £ cannot distinguish states S and S' , that 
observer must not be able to distinguish any execution E beginning 
with S from any execution E' beginning with S ' : 

(S «£ 5') ^E&eE' 

This property is too strong because it rules out two sources of non- 
determinism that we want to allow: first, the ability of any transac- 
tion to start at any time, and second, network delays. Therefore, we 
relax observational determinism to permit certain nondeterminism. 
We only require that executions be indistinguishable to the observer 
if their NIEs are indistinguishable to the observer: 

(S S' A NIE(E) NIE(E')) => E E' 

We call this relaxed property relaxed observational determinism. 
It might appear to be equivalent to observational determinism, but 
with the NIEs encoded in the start states. This is not the case. If 
NIEs were encoded in the start states, protocols would be able to 
read which transactions will start and when messages will arrive in 
the future. Therefore relaxed observational determinism captures 
something that observational determinism does not: unknowable 
but “allowed” nondeterminism at any point in an execution. 

By deliberately classifying start events and network delays as in- 
put, we allow certain kinds of information leaks that observational 
determinism would not. Specifically, a malicious network could 
leak information by manipulating the order or timing of message 
delivery. However, such a network could by definition communi- 
cate information to its co-conspirators anyway. Information can 
also be leaked through the order or timing of start events. This 
problem is beyond the scope of this work. 

Conditioning the premise of the security condition on the indis- 
tinguishability of information that is allowed to be released is an 
idea that has been used earlier [39], but not in this way, to our 
knowledge. 

In our hospital example, as illustrated in Fig. 4, the system de- 
termines which of Patsy’s transactions (the one with the dashed 
events, or the one without the dashed events) will run based on 
whether p.hasHiv is true. We can consider p.hasHiv’s value to 
be a high-security event that happens before all reads of p . hasHiv. 
If we classify this past high-security event as input, and all low- 
security events as low-observable for Mallory, then we must en- 
sure that when Patsy’s code runs, the low-security projections of 
resulting executions are always the same, regardless of whether 
p.hasHiv. Patsy’s possible transactions in Fig. 4 allow for ob- 
servational determinism, while her transactions in Fig. 3 do not, 
since whether or not Read address occurs depends on p.hasHiv. 
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Figure 9: Transactions that cannot be securely serialized. Dave’s 
transaction includes ro, ri, r 2 , and r 3 . Carol’s includes bo, bi, 
b 2 , and b 3 . Cloud providers Alice and Bob must decide how to 
order their events. Alice and Bob may not influence each other, and 
Carol and Dave may not influence each other, as represented by the 
wall. For these transactions to be serializable, Alice’s ordering of 
r 2 and b 2 must agree with Bob’s ordering of r 3 and b 3 . 


Whether or not the system actually maintains observational deter- 
minism, however, depends on the protocol scheduling the events. 

Def. 3 (Protocol Security). A protocol is considered se- 
cure if the set of resulting executions satisfies relaxed observational 
determinism for any allowed sets of information-flow secure trans- 
actions and any possible NIEs. 

4. IMPOSSIBILITY 

One of our contributions is to show that even in the absence of 
timing channels, there is a fundamental conflict between secure 
noninterference and serializability. Previous results showing such 
a conflict, for example the work of Smith et al. [42] consider only 
confidentiality and show only that timing channels are unavoidable. 

Theorem 1 (Impossibility). No secure protocol 3 can se- 
rialize all possible sets of information-flow secure transactions. 4 

We assume protocols cannot simply introduce an arbitrarily trusted 
third party; a protocol must be able to run using only the set of 
locations that have events being scheduled. 

PROOF, (by counterexample) Consider the counterexample shown 
in Fig. 9. Alice and Bob are both cloud computing providers who 
keep strict logs of the order in which various jobs start and stop. 
Highly trusted (possibly government) auditors may review these 
logs, and check for consistency, to ensure cloud providers are hon- 
est and fair. As competitors, Alice and Bob do not want each other 
to gain any information about their services, and do not trust each 
other to affect their own services. 

3 barring unforeseen cryptographic capabilities (§4.1) 

4 In fact, what we prove is stronger. Our proof holds for even pos- 
sibilistic security conditions [31], which are weaker than relaxed 
observational determinism (see supplemental technical report). No 
protocol whose resulting traces satisfy even this weaker condition 
can serialize all sets of information-flow secure transactions. 


Carol and Dave are presently running jobs on Alice’s cloud. Both 
Carol and Dave would like to stop their jobs on Alice’s cloud, and 
start new ones on Bob’s cloud. Each wants to do this atomically, 
effectively maintaining exactly one running job at all times. Carol 
and Dave consider their jobs to be somewhat confidential; they do 
not want each other to know about them. Unlike the example from 
Fig. 1, Dave and Carol’s transactions do not go through a third 
party like Rainforest. For the transactions to be serializable, Alice’s 
ordering of the old jobs stopping must agree with Bob’s ordering 
of the new jobs starting. 

These transactions feature at least 8 events: 
ro : Dave sends a message to Alice 

ri : Dave sends a message to Bob 

r 2 : Alice receives a message from Dave, ending a job. 

r 3 : Bob receives a message from Dave, beginning a job. 

b 0 : Carol sends a message to Alice 

bi : Carol sends a message to Bob 

b 2 : Alice receives a message from Carol, ending a job. 

b 3 : Bob receives a message from Carol, beginning a job. 

No events at Alice’s location should influence events at Bob’s 
location, and vice-versa. No events at Carol’s location should in- 
fluence events at Dave’s location, and vice-versa. 

Alice and Bob must each finish with ordered logs including job 
beginnings and endings. This means they must assign a happens- 
before (— >) relation to their events above. For these transactions 
to be serializable, Alice’s ordering of r 2 and b 2 must agree with 
Bob’s ordering of r 3 and b 3 . 

Lemma 1 . These transactions are information-flow secure. 
The two transactions in Fig. 9 are information-flow secure ( Def. 2). 

PROOF. The only happens-before relationships within transac- 
tions are for the sending and receipt of messages, explicitly carry- 
ing information readable to the recipient. All four are consistent 
with permitted information flows. □ 

Lemma 2. No protocol can securely serialize these transac- 
tions. Specifically, no protocol accepting these transactions can 
preserve possibilistic noninterference. 

PROOF. In any system with an asynchronous network, it is pos- 
sible to reach a state in which Carol’s message to Alice has arrived, 
but not her message to Bob, and Dave’s message to Bob has arrived, 
but not his message to Alice. In other words, events r 2 and b 3 have 
not yet occurred. Fig. 10 illustrates this situation. In this state, nei- 
ther Alice nor Bob can know whether one or both transactions have 
begun. It is impossible for either to communicate this information 
to the other without violating possibilistic noninterference. Specif- 
ically, any protocol that relayed such information from one cloud 
provider to the other would allow the recipient to distinguish the or- 
der of message delivery to the other cloud provider. That ordering 
is considered secret input, and so this would be a security violation. 
All executions with identical start states, and identical inputs visi- 
ble to Alice, but differently ordered network delay events at Bob, 
which are inputs invisible to Alice, would become distinguishable 
to Alice. Even possibilistic noninterference would therefore be vi- 
olated (§3.7.1). 

Additionally, we have assumed that a protocol must be able to 
schedule any subset of the allowed transactions’ start events. There- 
fore valid executions exist in which, say, only Carol’s transaction 
runs, so Alice receives only information about Carol’s transaction, 
and commits Carol’s transaction first. Therefore a valid execution 
must exist in which Alice commits Carol’s transaction first, before 
receiving any further input from Dave or Bob, and likewise, Bob 
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Figure 10: An intermediate state of an execution featuring the 
transactions from Fig. 9. 


commits Dave’s transaction first, without further input from Carol 
or Alice. Thus any protocol satisfying possibilistic noninterference 
can schedule inconsistently: the transactions cannot be securely se- 
rialized. □ 

Thus, with this scenario as a counterexample, no secure protocol 
can serialize all possible sets of information-flow secure transac- 
tions. 

4.1 Cryptography 

This essentially information-theoretic argument does not account 
for the possibility that some protocol could produce computation- 
ally indistinguishable traces that are low-distinguishable with suffi- 
cient computational power (e.g., to break encryption). However, we 
are unaware of any cryptographic protocols that would permit Al- 
ice and Bob to learn a consistent order in which to schedule events 
without learning each other’s confidential information. 

5. ANALYSIS 

Although secure scheduling is impossible in general, many sets 
of transactions can be scheduled securely. We therefore investigate 
which conditions are sufficient for secure scheduling, and what pro- 
tocols can function securely under these conditions. 

5.1 Monotonicity 

A relatively simple condition suffices to guarantee schedulabil- 
ity, while preserving relaxed observational determinism: 

Def. 4 (Monotonicity). A transaction is monotonic if it 
is information-flow secure and its events are totally ordered by 
happens -before (— >). 

Theorem 2 (Monotonicity => Schedulability). 

A protocol exists that can serialize any set of monotonic transac- 
tions and preserve relaxed observational determinism. 

PROOF. Monotonicity requires that each event must be allowed 
to influence all future events in the transaction. A simple, pes- 
simistic transaction protocol can schedule such transactions securely. 
In order to define this protocol, we need a notion of locks within our 
model. 


Locks. A lock consists of an infinite set of events for each al- 
lowed transaction. A transaction acquires a lock by scheduling any 
event from this set. It releases a lock by scheduling another event 
from this set. Thus, in a system state S, a transaction T holds a 
lock if S contains an odd number of events from the lock’s set cor- 
responding to T. No correct protocol should result in a state in 
which multiple transactions hold the same lock. All pairs of events 
in a lock conflict, so scheduled events that are part of the same lock 
must be totally ordered by happens-before (— >). All events in a 
lock share a location, which is considered to be the location of the 
lock itself. Likewise, all events in a lock share a label, which is 
considered to be the label of the lock itself. 

A critical property for transaction scheduling is deadlock free- 
dom [20, 41], which requires that a protocol can eventually sched- 
ule all events from any transaction whose start event has been sched- 
uled. A system enters deadlock when it reaches a state after which 
this is not the case. For example, deadlock happens if a protocol re- 
quires two transactions each to wait until the other completes: both 
will wait forever. If all transactions are finite sets of events (i.e., 
all transactions can terminate), then deadlock freedom guarantees 
that a system with a finite set of start events eventually terminates, 
a liveness property. Deadlock freedom is essential to distributed or 
parallel scheduling, but notoriously difficult to get right [41]. 

We now describe a deadlock-free protocol that can securely se- 
rialize any set of monotonic transactions, and preserve relaxed ob- 
servational determinism: 

• Each event in each transaction has a corresponding lock, ex- 
cept start events. 

• Any events that have the same label share a lock, and this 
lock shares a location with at least one of the events. Con- 
flicting events are assumed to share a label (§3.4). 

• A transaction must hold an event’s lock to schedule that event. 

• A transaction acquires locks in sequence, scheduling events 
as it goes. Since all events are ordered according to a global 
security lattice, all transactions that acquire the same locks 
do so in the same order. Therefore they do not deadlock. 

• If a lock is already held, the transaction waits for it to be 
released. 

• When all events are scheduled, the transaction commits, re- 
leasing locks in reverse order. Any messages sent as part of 
the transaction would thus receive a reply, indicating only 
that the message had been received, and all its repercussions 
committed. We call these replies commit messages. 

• For each location, the protocol rotates between all uncommit- 
ted transactions, scheduling any intermediate events (such as 
lock acquisitions) until it either can schedule one event in the 
transaction or can make no progress, and then rotates to the 
next transaction. 

Security Intuition. Acquiring locks shared by multiple events on 
different locations requires a commit protocol between those loca- 
tions. However, this does not leak information because all locations 
involved are explicitly allowed to observe and influence all events 
involved. Therefore several known commit protocols will do, in- 
cluding 2PC. Since the only messages sent as part of the protocol 
are commit messages, and each recipient knows it will receive a 
commit message by virtue of sending a message in the protocol, 
no information (other than timing) is transferred by the scheduling 
mechanism itself. 




Relaxed observational determinism. This protocol, implemented 
with monotonic transactions, satisfies relaxed observational deter- 
minism, our slightly relaxed version of observational determinism 
(§ 3 . 7 . 2 ). We consider an event observable to an observer with la- 
bel £ if the label of the event flows to £. For any two executions 
beginning with equivalent states (for some observer £), 

[ 0 ] 77 i[ 0 ] 

If the executions Eo and E\ have the same ^-observable inputs, 
which is to say transaction start events and network delay events, 
then the protocol requires Eo and E\ to be indistinguishable to L 
The observer of label £ can only observe a prefix of each transaction 
being scheduled in a round-robin fashion, and commit messages for 
each arriving sometime thereafter. Arrival time of these commit 
messages is considered an input, and so all events visible in Eo and 
Ei are deterministic results of the events visible in the start states, 
and the NIEs. Each distinct state in an execution, as observed at £, 
will be deterministically predicted by prior states and inputs. Thus 
relaxed observational determinism is preserved. 

Serializability. Transactions consist of totally ordered series of 
events. Let e\ be the first event in T\ conflicting with any event 
in Th. Let be the event in T2 with which e± conflicts. Suppose 
they are scheduled such that ei^>e2. Therefore all events in T2 
after and including e2 cannot be scheduled until T\ commits and 
releases its locks. No event in T 2 scheduled before e2 can conflict 
with an event in T\ after ei, by monotonicity, or before ei, by the 
definition of e\. Thus all conflicting events in T2 are scheduled 
after all events in T\ , so no event in T\ can happen after an event 
in T2. Therefore, this pessimistic protocol ensures serializability. 

Liveness. This scheduling system cannot result in deadlock, 
since all transactions acquire locks in strictly increasing order on 
the lattice, so any set of transactions that acquire the same locks 
must do so in the same order. 

Therefore, monotonicity is sufficient to guarantee secure schedu- 
lability. □ 

5.2 Relaxed Monotonicity 

Monotonicity, while relatively easy to understand, is not the weak- 
est condition we know to be sufficient for secure schedulability. It 
can be substantially relaxed. In order to explain our weaker condi- 
tion, relaxed monotonicity , we first need to introduce a concept we 
call visibility : 

Def. 5 (VlSlBLE-To). An event e in transaction T is visible 
to a location L if and only if it happens at L, or if there exists 
another event e E T at L, such that e—>e. 

Def. 6 (Relaxed Monotonicity). A transaction T satis- 
fies relaxed monotonicity if it is information-flow secure and for 
each location L, all events in T visible to L happen before all 
events in T not visible to L. 

In § 6 , we demonstrate that relaxed monotonicity guarantees schedu- 
lability. Specifically, we present a staged commit protocol, and 
prove that it schedules any set of transactions satisfying relaxed 
monotonicity, while preserving relaxed observational determinism 
(Thm. 4 ). 

5.3 Requirements for Secure Atomicity 

Monotonicity and relaxed monotonicity are sufficient conditions 
for a set of transactions to be securely schedulable. Some sets of 
transactions meet neither condition, but can be securely serialized 
by some protocol. For example, any set of transactions that each 
happen entirely at one location can be securely serialized if each 


location schedules each transaction completely before beginning 
the next. We now describe a relatively simple condition that is 
necessary for any set of transactions to be securely scheduled. 

Decision Events and Conflicting Events 

In order to understand this necessary condition, we first describe 
decision events and conflicting events. 

Borrowing some terminology from Fischer, Lynch, and Pater- 
son [ 21 ], for a pair of transactions T\ and T2, any system state is 
either bivalent or univalent. A system state is bivalent with respect 
to Ti and T2 if there exist two valid executions that both include 
that state, but end with opposite orderings of Ti and X2. A sys- 
tem state is univalent with respect to T\ and T2 otherwise: for one 
ordering of the transactions, no valid execution ending with that 
ordering contains the state. 

We can define a similar relationship for start events: for any pair 
of distinct start events s\ and S2, a system state is bivalent with re- 
spect to those events if it features in two valid executions, both of 
which have s 1 and S2 in scheduled transactions, but those transac- 
tions are in opposite order. A system state is univalent with respect 
to si and S2 otherwise. 

All full executions (i.e., those starting with an empty state) that 
order a pair of transactions begin in a bivalent state with respect to 
their start events, before either is scheduled. By our definition of se- 
rializability and transaction ordering, once transactions are ordered, 
they cannot be un-ordered. Any execution that orders the transac- 
tions therefore ends in a univalent state with respect to their start 
events. Any such execution consists of a sequence of 0 or more bi- 
valent states followed by a sequence of univalent states. The event 
that is scheduled in the first univalent state, in a sense, decides the 
ordering of the transactions. We call it the decision event. 

We call any event in Ti or T2 that conflicts with an event in the 
other transaction a conflicting event. 

Lemma 3 (Decision Event -> Conflicting Events). 
For any univalent state S with Ti—>T 2, there exists a full execution 
E ending in S featuring a decision event e d that happens before 
(—>) all conflicting events in Ti and T2 ( other than e d itself if e d is 
a conflicting event). 

PROOF. Assume the contradiction. Then for any full execution 
E' ending in S, an equivalent execution exists featuring a state in 
which a conflicting event e c is scheduled, but the decision event of 
E' is not. Such an equivalent execution would by definition have 
a different decision event, since e c ’s presence in a state makes the 
state univalent. By our assumption, this equivalent execution has 
conflicting events that neither are, nor happen after, its decision 
event. This implies yet another equivalent execution with yet an- 
other state featuring an even earlier conflicting event but not the 
decision event, and so on. Since all states are finite sets, and ^is 
a strict partial order, this infinite descending chain is impossible. 
There must exist an execution E ending with S with decision event 
ed that happens before all conflicting events in T\ and T2. □ 

We show that two fundamental system state properties are nec- 
essary for secure scheduling: 

Def. 7 (First-Precedes-Decision). State S satisfies First- 
Precedes-Decision if for any pair of transactions T\ and T2 in S 
with Ti —>T2, there is a full execution E ending in S with a decision 
event e d that either is in T\, or happens after an event in T±. 

Def. 8 (Decision-Precedes-Second). A state S satisfies 
Decision-Precedes-Second if for any pair of transactions Ti and 
T2 in S with T\—>T2, there is a full execution E' ending in S with 
a decision event e' d , such that no event in T2 happens before e' d . 
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Therefore, for a protocol to be secure, it must ensure resulting 
system states have these properties. 

Theorem 3 (Necessary Condition). Any secure, dead- 
lock-free protocol p must ensure that all full executions consistent 
with p feature only states satisfying both First-Precedes-Decision 
and Decision-Precedes-Second. 

PROOF. Given T±—>T 2 , any execution E' ending in S features a 
decision event e d . Decision events for the same pair of transactions 
in equivalent executions must agree on ordering, by the definition 
of equivalent execution. If T\ does not contain E’s decision event, 
e d , or any event that happens before e d , then there exists an equiv- 
alent execution in which e d is scheduled before any events in Ti 
or T 2 . This execution would imply the existence of a system state 
in which no event in either transaction is scheduled, but it is im- 
possible to schedule T 2 before Ti, regardless of inputs after that 
state. If, after this state, the start event for T 2 were scheduled, but 
not the start event for T ± , then T 2 cannot be scheduled. This con- 
tradicts a the deadlock-freedom requirement: no protocol should 
result in a system state in which a supported transaction can never 
be scheduled. 

Therefore some event in T\ either is or happens before e ( j for 
some full execution E ending in S. 

If T\ and T 2 conflict, then e d either is an event in T\ or happens 
before an event in Ti, by Lemma 3. If an event e 2 E T 2 happens 
before e d , then either e d E Ti, and 

e2—>e d => T 2 —>Ti 

which is impossible, by the definition of happens-before, or 

3d E T\.e d — >ei, and 

e2^e^^ei => e2^ei => T2—>T\ 

which is also impossible, by the definition of happens-before. 

If T\ and T 2 do not conflict, then the only way T\ ->T 2 implies 
that there exists some chain Ti^T 3 ^T 4 ^> . . . —>T n —>T 2 such that 
and each transaction in the chain conflicts with the next. Therefore, 
by the above proof, an equivalent execution exists in which each 
transaction in the chain contains the decision event for ordering it- 
self and the following transaction, and no events in the following 
transaction are before that decision event. 

Therefore there exists some equivalent execution E' in which 
no event in T 2 happens before the decision event e d deciding the 
ordering between T\ and T 2 . □ 

Although Thm. 3 may seem trivial, it represents some impor- 
tant conclusions: No protocol can make any final ordering decision 
until at least one transaction involved has begun. Furthermore, it is 
impossible for the later transaction to determine the decision. Truly 
atomic transactions cannot include any kind of two-way interaction 
or negotiation for scheduling. 

6. THE STAGED COMMIT PROTOCOL 

We now present the staged commit protocol (SC) and prove that 
it is secure, given transactions satisfying relaxed monotonicity. 

SC is a hybrid of traditional serialization protocols, such as 2PC, 
and the simple pessimistic protocol described in the proof of Thm. 2. 
Compared to our simple pessimistic protocol, it allows a broader 
variety of transactions to be scheduled (relaxed monotonicity vs. 
regular monotonicity), which in turn allows more concurrency. A 
transaction is divided into stages , each of which can be securely 
committed using a more traditional protocol. The stages them- 
selves are executed in a pessimistic sequence. 


Each event scheduled is considered to be either precommitted 
or committed. We express this in our model by the presence or 
absence of an “isCommitted” event corresponding to every event 
in a transaction. Intuitively, a precommitted event is part of some 
ongoing transaction, so no conflicting events that happen after a 
precommitted event should be scheduled. A committed event, on 
the other hand, is part of a completed transaction; conflicting events 
that happen after a committed event can safely be scheduled. Once 
an event is precommitted, it can never be un-scheduled. It can only 
change to being committed. Once an event is committed, it can 
never change back to being precommitted. 

• The events of each transaction are divided into stages. Each 
stage will be scheduled using traditional 2PC, so aborts within 
a stage will be sent to all locations involved in that stage. 

To divide the events into stages, we establish equivalence 
classes of the events’ labels. Labels within each class are 
equivalent in the following sense: when events with equiva- 
lent labels are aborted, those aborts can securely flow to the 
same set of locations. An event’s abort can always flow to the 
event’s own location, so locations involved in a stage can se- 
curely ensure the atomicity of the events in that stage. Since 
conflicting events have the same security labels, they will 
be in the same equivalence class. We call these equivalence 
classes conflict labels (cl). 

• Each stage features events of the same conflict label, and is 
scheduled with 2PC. One location must coordinate the 2PC. 
All potential aborts in the stage must flow to the coordina- 
tor, and some events on the coordinator must be permitted to 
affect all events in the stage. Relaxed monotonicity implies 
that at least one such location exists for each conflict label. 

When a stage tries to schedule an event, but finds a precom- 
mitted conflicting event, it aborts the entire stage. Because 
conflicting events have the same label, these aborts cannot 
affect events on unpermitted locations. 

When a stage’s 2PC completes, the events in the stage are 
scheduled, and considered precommitted. 

• Each transaction precommits its stages as they occur. To 
avoid deadlock, we must ensure that whenever two transac- 
tions feature stages with equal conflict labels, they precom- 
mit those stages in the same order. Therefore, the staged 
commit protocol assumes an ordering of conflict labels. This 
can be any arbitrary ordering, so long as (1) it totally orders 
the conflict labels appearing in each transaction, and (2) all 
transactions agree on the ordering. 

• When all stages are precommitted, all events in the trans- 
action can be committed. Commit messages to this effect 
are sent between locations, backwards through the stages. 
Whenever an event in one stage triggers an event in the next, 
the locations involved can be sure a commit message will take 
the reverse path. The only information conveyed is timing. 

Because events in a precommitted stage cannot be un-scheduled 
or “rolled back”, a participant that is involved only in an earlier 
stage is prevented from gleaning any information about later stages. 
The participant will only learn, eventually, that it can commit. 

Patsy’s transaction in Fig. 4c has at least two stages when the 
patient has HIV: 
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1 . Patsy begins the transaction (Patsy start), and reads the ad- 
dress (Read Address). This stage will be atomically precom- 
mitted, and this precommit process will determine the rela- 
tive ordering of Patsy’s transaction and Mallory’s, indepen- 
dent of more secret events. 

2 . Patsy finds that the patient has HIV (Read HIV), and prints 
the patient’s address (Print address). 

Theorem 4 (Security of SC). Any set of transactions sat- 
isfying relaxed monotonicity are serialized by SC securely without 
deadlock. 

Proof. 

Security. SC preserves relaxed observational determinism. Intu- 
itively, any information flows that it adds are already included in 
the transaction. 

SC adds no communication affecting security: 

• Communication within each stage is strictly about events that 
all participants can both observe. 

• For each pair of consecutive stages, at least one participant 
from the first stage can notify a participant in the second 
stage securely, when it is time for the second stage to be- 
gin. Relaxed monotonicity ensures the second stage contains 
an event that happens after an event in the first stage, repre- 
senting a line of communication. 

• Communication for commits can safely proceed in reverse 
order of stages. Within each stage, each participant can se- 
curely forward a commit message to all other participants. 
Between stages, commit messages can be sent back along the 
same channels used to notify each stage the previous one had 
precommitted. Each participant knows when it precommits 
exactly which commit messages it will receive. The commit 
messages themselves do not leak any information (other than 
timing) to their recipients. 

Therefore SC adds no unauthorized information flows. 

Specifically, for any given participant’s label i, events within a 
stage visible to i are scheduled deterministically based only on in- 
formation visible to L Commit messages (and affiliated events) for 
visible stages arrive eventually, at a time determined by network 
delay events, which we consider input. Other stages’ events are not 
observable to L 

Therefore, for any two executions beginning with states indistin- 
guishable to £, with NIEs visible to £, all scheduled events visible 
to i would be indistinguishable. Thus relaxed observational deter- 
minism is preserved. 

Serializability. Any set of transactions with relaxed monotonic- 
ity scheduled by SC will be serializable. 

Lemma 4 (Precommitted Snapshot). 

Any execution in which an event in a transaction is committed 
features a system state in which all events in the transaction are 
precommitted. 

Proof. Stages are totally ordered, and each waits until the final 
stage commits before (— >) any of its events commit. The final stage 
precommits before (— >) it commits, and so there is a system state 
in which all events in the transaction are precommitted. □ 

Let E be an execution where any two conflicting transactions T\ 
and T2 both have at least one event that commits. Given Lemma 4 , 
E must feature two states: one in which all events in T\ are precom- 
mitted, and another in which all events of T2 are precommitted. As 


T\ and T2 conflict, these states cannot be the identical. (An event 
is never scheduled while a conflicting event is precommitted.) 

One transaction must be scheduled before (— >) the other. With- 
out loss of generality, let it be T\ . No equivalent execution can fea- 
ture a state in which an event in T2 is scheduled before an event in 
Ti , as this would require a conflicting event in T2 to be precommit- 
ted before its corresponding conflicting event in Ti is committed. 
The corresponding conflicting event in T\ must be precommitted 
before any event in T\ commits, and we require that all events in 
q2 remain precommitted until after an event in T± commits. 

Therefore, if T\->T2 then it is impossible for T2— >Ti. Thus 
SC guarantees a strict partial order of transactions, and therefore 
serializability. 

Deadlock Freedom. 

A deadlock can occur only if there is a cycle of dependencies 
among transactions, in which transaction T\ depends on T 2 if and 
only if T2 has precommitted an event conflicting with an unsched- 
uled event in T\ . 

Conflicting events share labels, and stages are defined by labels. 
All transactions must therefore order the stages of conflicting pairs 
in the same way. One event can only ever depend on an event in 
its own or in a prior stage. Stages are precommitted in order, so no 
dependency cycle featuring events in different stages is possible. 

Each stage is precommitted atomically using 2 PC. 2 PC preserves 
deadlock freedom, meaning no cycle featuring only events in the 
same stage is possible. 

Therefore no cycles, and thus no deadlock, can exist with SC. 

SC is secure, deadlock-free, and guarantees serializability when 
the transactions have relaxed monotonicity. 

The Importance of Optimism 

SC specifies only a commit protocol. Actual computation (which 
generates the set of events) for each transaction can be done in 
advance, optimistically. If one stage precommits and the next is 
blocked by a conflicting transaction, optimistically pre-computed 
events would have to be rolled back. However, no precommit- 
ted event need be rolled back. In fact, it would be insecure to do 
so. Thus SC allows for partially optimistic transactions with partial 
rollback. 

Our model requires only that a transaction be a set of events. In 
many cases, however, it is not possible to know which transaction 
will run when a start event is scheduled. For example, a transaction 
might read a customer’s banking information from a database and 
contact the appropriate bank. It would not be possible to know 
which bank should have an event in the transaction beforehand. If 
a system attempted to read the banking information prior to the 
transaction, then serializability is lost: the customer might change 
banks in between the read and the transaction, and so one might 
contact the wrong bank. 

Optimism solves this problem: events are pre-computed, and 
when an entire stage is completed, that stage’s 2 PC begins. This 
means that optimism is not just an optimization; it is required for 
secure scheduling in cases where the transactions’ events are not 
known in advance. 

7. IMPLEMENTATION 

We extended the Fabric language and compiler to check that 
transactions can be securely scheduled, and we extended the Fabric 
runtime system to use SC. Fabric and IFDB [ 40 ] are the two open- 
source systems we are aware of that support distributed transac- 
tions on persistent, labeled data with information flow control. Of 
these, we chose Fabric for its static reasoning capabilities. IFDB 
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1 atomic { 

PC Possible conflictors 

2 

String{7} p = post.read(); 

A 

{Alice, Bob, Carol} 

3 

Comments!/} c; 

A 

- 

4 

if (p. contains(”f izz")) { 

_L 

- 

5 

c.write(”buzz") ; 

£ 

{Alice, Carol} 

6 

if (p. contains(”buzz")) { 

L 

- 

7 

c. write("f izz") ; 

£ 

{Alice, Carol} 


8 } 

9 } 

Figure 11: Carol’s program in our Blog example: Carol reads a 
post with label £, and depending on what she reads, writes a com- 
ment with label Label i permits Alice, Bob, and Carol to read 
the post, while l' keeps the Comments more private and allows only 
Alice and Carol to view or edit. 

checks labels entirely dynamically, so it cannot tell if a transaction 
is schedulable until after it has begun. 

7.1 The Fabric Language 

The Fabric language is designed for writing distributed programs 
using atomic transactions that operate on persistent, Java-like ob- 
jects [30], It has types that label each object field with information 
flow policies for confidentiality and integrity. The compiler uses 
these labels to check that Fabric programs enforce a noninterfer- 
ence property. However, like all modern systems built using 2PC, 
Fabric does not require that transactions be securely scheduled ac- 
cording to the policies in the program. Consequently, until now, 
abort channels have existed in Fabric. 

We leverage these security labels and extend the compiler to ad- 
ditionally check that transactions in a Fabric program are mono- 
tonic (§5). This implementation prevents confidentiality breaches 
via abort channels. Preventing integrity breaches would require 
further dynamic checks, which we leave to future work. 

7.2 Checking Monotonicity 

Our modification to the Fabric compiler enforces relaxed mono- 
tonicity (Def. 6). Our evaluation (§ 8) shows that enforcing this 
condition does not exclude realistic and desirable programs. Our 
changes to the Fabric compiler and related files include 4.1k lines 
of code (out of roughly 59k lines). 

7 . 2.7 Events and Conflict Labels in Fabric 

The events in the system model (§3) are represented in our im- 
plementation by read and writes on fields of persistent Fabric ob- 
jects. The label of the field being read or written corresponds to the 
event labels in our model. 

SC (§6) divides events into stages based on conflict labels (cl). 
In our implementation, we define the cl of an event e to correspond 
to the set of principals authorized to read or write the field that is 
being accessed by e. If e is a write event, this set contains ex- 
actly those principals that can perform a conflicting operation (and 
thereby receive an abort); if e is a read event, the set is a conserva- 
tive over- approximation, since only the writers can conflict. 

Fig. 1 1 presents a program in which Carol schedules two events 
within a single transaction. First, she reads a blog post with security 
label 7. Second, she writes a comment (whose content depends 
on that of the post) with label £'. Since £ permits Alice, Bob, or 
Carol to read the post, the cl of the first event includes all three 
principals. However, only Alice and Carol can read or write the 
comment, so when Carol goes to write it, only Alice or another 
transaction acting on behalf of Carol could cause conflicts. The cl 
of the write therefore includes only Alice and Carol. 

7 . 2.2 Program Counter Label 


The program counter label (pc) [16] labels the program context. 
For any given point in the code, the pc represents the join (least 
upper bound) of the labels of events that determine whether or not 
execution reaches that point in the code. These events include those 
occurring in if- statement and loop conditionals. For instance, in 
Fig. 11, whether line 5 runs depends on the value of p, which has 
label £. Therefore, the fact that line 5 is executing is as secret as p, 
and the pc at line 5 is £. 

SC requires that when events with the same cl are aborted, those 
aborts can securely flow to the same set of locations. When an 
event causes an abort, the resulting abort messages carry informa- 
tion about the context in which the event occurs. Therefore, we 
enforce the requirement by introducing a constraint on the program 
context in which events may occur: the pc must flow to the princi- 
pals in the conflict label. 

pc □ cl (1) 

Eliding the details of how Fabric’s labels are structured, in Fig. 11, 
A flows to everything, and 7, the label of the blog post, does flow 
to the conflict label, indicating that both Alice and Carol can cause 
a conflict. Therefore, Eqn. (1) holds on lines 2, 5, and 7. 

7.2.3 Ordering Stages 

Each stage consists of operations with the same cl. To ensure 
all transactions precommit conflicting stages in the same order, we 
adopt a universal stage ordering: 

principals (cl*) 2 principals (cli+i) (2) 

The set of principals in each stage must be a strict superset of the 
principals in the next one. This ensures that unrestricted infor- 
mation can be read in one stage and sensitive information can be 
modified in a later stage in the same transaction. In the hospital 
example (Fig. 4), Read HIV has a conflict label that only includes 
trusted personnel, while Read address has a conflict label that in- 
cludes more hospital staff. As a result, our implementation requires 
that Read address be staged before Read HIV in Patsy’s transaction. 

In Fig. 11, our stage ordering means that the read on line 2, with 
a cl of {Alice, Bob , Carol} belongs in an earlier stage than the 
write, which features a cl of only {Alice, Carol}. 

7.2.4 Method Annotations 

To ensure modular program analysis and compilation, each method 
is analyzed independently. Fabric is an object-oriented language 
with dynamic dispatch, so it is not always possible to know in 
advance which method implementation a program will execute. 
Therefore, the exact conflict labels for events within a method call 
aren’t known at compile time. In order to ensure each atomic pro- 
gram can divide into monotonic stages, we annotate each method 
with bounds on the conflict labels of operations within the method. 
These annotations are the security analogue of argument and return 
types for methods. 

7.3 Implementing SC 

We extended the Fabric runtime system to use SC instead of tra- 
ditional 2PC, modifying 2.4k lines of code out of a total of 24k lines 
of code in the original implementation. Specifically, we changed 
Fabric’s 2PC-based transaction protocol so that it leaves each stage 
prepared until all stages are ready, and then commits. 

Since Fabric labels can be dynamic, the compiler statically deter- 
mines potential stagepoints — points in the program that may begin 
a new stage — along with the conflict labels of the stages immedi- 
ately surrounding the potential stagepoint. If the compiler cannot 
statically determine whether the conflict labels before and after a 
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stagepoint will be different, it inserts a dynamic equivalence check 
for the two labels. At run time, if the two labels are not equivalent, 
then a stage is ending, and the system precommits all operations 
made thus far. To precommit a stage, we run the first (“prepare”) 
phase of 2PC. If there is an abort, the stage is re-executed until it 
eventually precommits. 

In Fig. 11, there is a potential stagepoint before lines 4 and 6, 
where the next operation in each case will not include Bob as a 
possible conflictor. The conflict labels surrounding the potential 
stagepoint are {Alice, Bob, Carol} (from reading the post on line 
2) and {Alice, Carol} (from writing the comment on either line 
4 or 6). If another transaction caused the first stage to abort, then 
Carol’s code would rerun up to line 4 or 6 until it could precommit, 
and then the remainder of the transaction would run. 

8. EVALUATION 

To evaluate our implementation, we built three example Fabric 
applications, and tested them using our modified Fabric compiler: 

• an implementation of the hospital example from §2; 

• a primitive blog application (from which Fig. 1 1 was taken), 
in which participants write and comment on posts with pri- 
vacy policies; and 

• an implementation of the Rainforest example from § 2. 

8.1 Hospital 

We implemented the programs described in our hospital example 
(Fig. 3). In the implementation, Patsy’s code additionally appends 
the addresses of HIV-positive patients to a secure log. In a third 
program, another trusted participant reads the secure log. 

With our changes, the compiler correctly rejects Patsy’s code. 
We amended her code to reflect Fig. 4. Of the 350 lines of code, 
we had to change a total of 1 13 to satisfy relaxed monotonicity and 
compile. Of these 113 lines, 23 were additional method annotations 
and the remaining 90 were the result of refactoring the transaction 
that retrieves the addresses of HIV-positive patients. SC scheduled 
the transactions without leaking information. The patient’s HIV 
status made Mallory neither more nor less likely to receive aborts. 

8.2 Blog 

In our primitive blog application, a store holds API objects, each 
of which features blog posts (represented as strings) with some se- 
curity label, and comments with another security label. These la- 
bels control who can view, edit, or add to the posts and comments. 

In one of our programs, the blog owner atomically reads a post 
and updates its text to alternate between “fizz” and “buzz”. In an- 
other program, another user comments on the first post (Fig. 11). 
To keep this comment pertinent to the content of the post, reading 
the post and adding the comment are done atomically. Since posts 
and comments have different labels, this transaction has at least two 
stages. In the first, the post is read, and in the second, the comment 
is written. 

We were able to compile and run these programs with our mod- 
ified system with relatively few changes. Of the 352 lines of code, 
we had to change a total of 50, primarily by adding annotations to 
method signatures (§7.2.4). 

8.3 Rainforest 

We implemented the Rainforest example from §2.1. In our code, 
two nodes within Rainforest act with Rainforest’s authority. They 
perform transactions representing the orders of Gloria and Fred 
from Fig. 1. Each transaction updates inventory data stored at one 
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Figure 12: Example policies for the Rainforest application. 


location, and banking data stored at another. Fig. 12 gives examples 
of the policies for price, inventory, and banking data. 

While attempting to modify this code to work with SC, we dis- 
covered that the staging order chosen in § 7.2.3 makes it impossi- 
ble to provide the atomicity of the original application while both 
meeting its security requirements and ensuring deadlock freedom. 

To illustrate, suppose Gloria is purchasing an item from Outel. 
To ensure she is charged the correct price, the event that updates 
the inventory must share a transaction with the one that debits Glo- 
ria’s bank account. The conflict label for the inventory event cor- 
responds to {Outel}, whereas the conflict label for the debit event 
corresponds to {Bank, Gloria}. Since neither is a subset of the 
other, the compiler cannot put them in the same transaction. 

These difficulties in porting the Rainforest application arise be- 
cause Fabric is designed to be an open system, and so an a pri- 
ori choice of staging order must be chosen. If the application 
were written as part of a closed system, deadlock freedom can be 
achieved by picking a staging order that works for this particular 
application (e.g., {Outel} before {Bank, Gloria}), but it might 
be difficult to extend the system with future applications. 

8.4 Overhead 

The staged commit protocol adds two main sources of overhead 
compared to traditional 2PC. First, each stage involves a round trip 
to prepare the data manipulated during the stage, leading to over- 
head that scales with the number of stages and with network la- 
tency. Second, as described in § 7.3, dynamic labels result in po- 
tential stagepoints, which must be resolved using run-time checks. 
The number of checks performed depends on how well the com- 
piler’s static analysis predicts potential stagepoints. 

We measured this overhead in our implementation on an Intel 
Core i7-2600 machine with 16 GiB of memory, using the transac- 
tions in our examples. The post and comment transactions in the 
blog example were each run continually for 15 minutes, and Patsy’s 
transaction in the hospital example was run continually for 1 hour. 

Fig. 13 gives the overall execution times for both the original 
system and the modified system. For the modified system, it also 
shows the number of stages for each transaction and the average 
time spent in dynamic checks for resolving potential stagepoints. 
The comment transaction in our experiments has one more stage 
than as described in Fig. 11, because in all transactions, there is 
an initial stage performed to obtain the principals involved in the 
application. 

By running the nodes on a single machine and using in-memory 
data storage, we maximize the fraction of the transaction run time 
occupied by dynamic checks. Nevertheless, this fraction remains 
small. While the effective low latency of communication between 
nodes reduces the overhead due to communication round-trip s for 
staging precommits, we report the number of stages, from which 
this overhead can be calculated for arbitrary latency. 

9. RELATED WORK 

Various goals for atomic transactions, such as serializability [34] 
and ACID [23], have long been proposed and widely studied, and 
are still an active research topic [36, 25, 42, 30, 8, 12]. While much 
of the recent interest has been focused on performance [18, 28, 44, 
3, 47, 45], we focus on security. 
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Figure 13: Performance overhead of SC. Reported times are per-transaction averages, across three 5-minute runs of the blog application and 
three 20-minute runs of the hospital application. Relative standard error of all measurements is less than 2%. 


Information leaks in commonly used transaction scheduling pro- 
tocols have been known for at least two decades [42, 7]. Kang 
and Keefe [24, 25] explore transaction processing in databases with 
multiple security levels. Their work, however, focuses on a set- 
ting with a global, trusted transaction manager and data replicated 
across whichever databases are permitted to read it. They also 
assume each transaction has a single security level, and can only 
“read down” and “write up.” Our work addresses a more general 
model. We require no fully trusted party and support transactions 
with more complex information flows. 

Smith et al. [42] show that strong atomicity, isolation, and con- 
sistency guarantees are not possible for all transactions in a gener- 
alized multilevel secure database. They propose weaker guarantees 
and give three different protocols that meet various weaker guar- 
antees. Their Low-Ready- Wait 2PL protocol is similar to SC, and 
provides only what the authors call ACIS - -correctness. Specifi- 
cally, “aborted operations at a higher level may prevent all lower 
level operations from beginning” [42, p37]. Although our imple- 
mentation is conservative and would not allow such a thing, the 
theory behind SC could allow a later stage with less trustworthy 
participants to hold up earlier, precommitted stages indefinitely. 
Duggan and Wu [19] observe that aborts in high-security subtrans- 
actions can leak information to low- security parent transactions. 
Their model of a single, centralized multilevel secure database with 
strictly ordered security levels is more restrictive than our distributed 
model and security lattice. Our abort channels generalize their ob- 
servation. They arrive at a different solution, building a theory of 
secure nested transactions. Atluri, Jajodia, and George [6] describe 
a number of known protocols requiring weaker guarantees or a sin- 
gle trusted coordinator. Our work instead focuses on securely se- 
rializing transactions in a fully decentralized setting. Our analysis 
is also the first in this vein to consider liveness: SC can guarantee 
deadlock freedom of transactions with relaxed monotonicity. 

In this work, we build on a body of research that uses lattice- 
based information flow labels and language-based information flow 
methods [15, 17, 38]. Relatively little work has studied informa- 
tion flow in transactional systems. Our implementation is built on 
Fabric [30, 4], a distributed programming system that controls in- 
formation flow over persistent objects. The only other information- 
flow-sensitive database implementation appears to be IFDB [40], 
which also does not account for abort channels. 

10. CONCLUSION 

There is a fundamental trade-off between strong consistency guar- 
antees and strong security properties in decentralized systems. We 
investigate the secure scheduling of transactions, a ubiquitous build- 
ing block of modern large-scale applications. Abort channels present 
a stark example of an unexplored security flaw: existing transac- 
tion scheduling mechanisms can leak confidential information, or 
allow unauthorized influences of trusted data. While some sets of 
transactions are impossible to serialize securely, we demonstrate 
the viability of secure scheduling. 

We present relaxed monotonicity, a simple condition, under which 
secure scheduling is guaranteed possible. Our staged commit pro- 
tocol can securely schedule any set of transactions with relaxed 


monotonicity, even in an open system. To demonstrate the practi- 
cal applicability of this protocol, we adapted the Fabric compiler 
to check transactional programs for conditions that allow secure 
scheduling. These checks are effective: the compiler identifies an 
intrinsic security flaw in one program, and accepts other, secure 
transactions with minimal adaptations. 

This work sheds light on the fundamentals of secure transac- 
tions. However, there is more work to be done to understand the 
pragmatic implications. We have identified separate necessary and 
sufficient conditions for secure scheduling, but there remains space 
between them to explore. Ultimately, abort channels are just one in- 
stance of the general problem of information leakage in distributed 
systems. Similar channels may exist in other distributed settings, 
and we expect it to be fruitful to explore other protocols through 
the lens of information flow analysis. 
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